Skip to content

Add read-only reader sites#116

Merged
colinmollenhour merged 7 commits into
mainfrom
megamind/read-only-readers
Jul 18, 2026
Merged

Add read-only reader sites#116
colinmollenhour merged 7 commits into
mainfrom
megamind/read-only-readers

Conversation

@colinmollenhour

@colinmollenhour colinmollenhour commented Jul 18, 2026

Copy link
Copy Markdown
Collaborator

AI Megamind - By: OpenCode / openai/gpt-5.6-sol

Closes #115.

Summary

  • add first-class read-only sites with conditional placement validation, per-site MySQL/Service controls, reader lag thresholds, generated CRDs, and chart synchronization
  • enforce non-promotable authority, direct-primary follower source convergence with pre/post-STOP GTID gates, backup/donor exclusions, reader-isolated readiness, endpoint shedding, internal administrative Services, and safe N-site rollouts
  • extend the default playground to a dedicated third reader worker and add release-profile reader data-loss/reclone coverage plus direct-source planned-switchover assertions
  • document topology, API, operations, monitoring, logs, backups, exposure, rollout, and clone impact

Operational impact

Existing primary-candidate and dr-only manifests retain their required lbIP/taintNodeSelector contract. Reader failures remain visible per site but do not degrade the failover group's writer readiness. Client site Services are now MySQL-only; operator and sidecar traffic uses an always-ClusterIP -internal Service. Existing Deployments migrate to per-site ConfigMaps through the ordered updater.

Test plan

  • make generate && make manifests — passed
  • diff -r config/crd/bases charts/bloodraven/crds — passed
  • make vet / make lint — passed
  • make test / make test-envtest — passed
  • operator, sidecar, plugin, and playground-chaos builds — passed
  • helm lint charts/bloodraven — passed
  • docs build and llms verification — passed
  • playground shell syntax and git diff --check — passed
  • hosted E2E workflow dispatched with profile=release; local context lacks Bloodraven CRDs/MFG

Megamind artifacts

  • Final plan: .tmp/megamind-read-only-readers/plans/final.md
  • Validated review findings: .tmp/megamind-read-only-readers/reviews/validated-findings.md
  • Latest fixed review: .tmp/megamind-read-only-readers/reviews/fixed-review-2.md
  • Local gates: .tmp/megamind-read-only-readers/final/local-gates.md

The .tmp run artifacts remain local/ignored; paths are included for maintainers with workspace access.

Megamind Educational Brief

Journey

Issue #115 began with useful primitives—centralized promotion eligibility, clone recovery, per-site status, and replica Services—but the planning critique showed that a healthy replication thread could still be chained through the demoted primary. Three models refined the task, and a three-round architecture debate converged unanimously on eight bundled decisions. Implementation was split into core operator, playground E2E, and docs scopes.

The first ultra-review validated 13 behavioral gaps, including destructive recovery after a replica-status probe error, recovery trusting a writable non-promotable site, stale serving labels under ambiguous authority, unsafe startup Deployment patching, and incomplete active-last verification. Targeted fixes plus two fixed-review passes closed all 13 before delivery.

Design decisions

Decision Why Rejected alternative
Separate periodic source convergence Running threads do not prove the follower uses the active primary; pre/post-STOP GTID checks close the applier race. Reusing history-dependent old-primary recovery and RESET REPLICA ALL.
Split client and internal Services Reader endpoints must shed when unhealthy while recovery and sidecar traffic remain routable. Exposing sidecar through NodePort/LoadBalancer or making control traffic depend on a health-gated Service.
Per-site ConfigMaps and serialized rollout Site overrides need isolated rendered hashes and crash-safe migration. A shared multi-key ConfigMap with an equivalent migration hazard.
Preserve all-site sidecar peers Maintains candidate/dr-only compatibility and topology-relay evidence. Reader-only exclusion that does not fix the same concern for other peers.
Deterministic PVC-loss E2E Reliably exercises empty-reader clone recovery on local storage. Literal local-PV node deletion, which can leave pods permanently pending.

Architecture

  • A valid authority is exactly one directly confirmed writable primary-candidate; any second writable site invalidates authority.
  • Every read-only follower with replication metadata periodically converges to that authority's internal Service using GTID containment before and after STOP, then CHANGE/START/verify.
  • Reader clients use a MySQL-only, healthy=yes-selected Service. Operator, replication, clone, backup, archiver, and sidecar traffic use the always-ClusterIP internal Service.
  • Reader serving health requires read-only state, running IO/SQL threads, direct converged source, known lag, and threshold compliance. Reader failures remain per-site signals and do not degrade writer readiness; anomalous writability remains a degraded fencing event.
  • Exact rendered config, peer membership, and render version feed per-site pod hashes. Safe followers update first; active drift requires a promotable handoff and strict confirmation that the demoted site follows the new primary.

Lessons

  • A non-promotable role still needs a call-site audit across authority, DNS, fencing, taints, donors, backups, rollout, labels, and readiness.
  • Observation failure is not evidence of absent replication; probe failures must interlock destructive recovery.
  • A health-gated data plane needs an unsheddable control plane.
  • Migration safety is strongest when level-driven: create desired resources, inspect uncached references, retain legacy state on uncertainty, and delete only after proof.
  • Hash rendered runtime dependencies such as peer addresses, not only raw user configuration.
  • Chaos scenarios should be reviewed as proofs, including whether API observation failures create blind spots.

Evidence

Grounded in issue #115, commit 922745ed73c1f52b001d5f1448b82f3f32fcba24, the changed API/controller/playground/docs files, all required local gate outcomes above, and the review/fix artifacts listed under Megamind artifacts. Hosted release-profile E2E was dispatched; this description intentionally does not claim it passed until the workflow completes.

Summary by CodeRabbit

  • New Features
    • Added support for non-promotable read-only reader sites and dedicated reader endpoints.
    • Added configurable replication-lag thresholds for reader traffic.
    • Added per-site MySQL configuration and Service overrides, including traffic policy, ports, types, and annotations.
    • Added source-convergence status and Prometheus metrics for follower replication health.
    • Added safer ordered multi-site updates, automatic reader recovery, and cold reclone workflows.
    • Expanded backup-source selection to exclude read-only readers.
  • Documentation
    • Updated configuration, failover, backup, monitoring, operations, and plugin guidance for reader and convergence behavior.
  • Tests
    • Expanded unit, integration, environment, and chaos coverage for multi-site and reader scenarios.

Copilot AI review requested due to automatic review settings July 18, 2026 03:06
@coderabbitai

coderabbitai Bot commented Jul 18, 2026

Copy link
Copy Markdown

Review Change Stack

Warning

Review limit reached

@colinmollenhour, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 3 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: f0c553e4-c7ca-418c-a867-2045254ed50a

📥 Commits

Reviewing files that changed from the base of the PR and between d8d689c and b7b292d.

📒 Files selected for processing (36)
  • cmd/playground-chaos/reset.go
  • cmd/playground-chaos/reset_test.go
  • docs/docs/monitoring-prometheus.mdx
  • docs/docs/monitoring.mdx
  • docs/docs/playground.mdx
  • internal/controller/reclone_test.go
  • internal/controller/reconciler.go
  • internal/controller/reconciler_test.go
  • internal/controller/runner.go
  • internal/controller/runner_test.go
  • internal/controller/source_convergence.go
  • internal/controller/source_convergence_test.go
  • internal/controller/topology.go
  • internal/controller/topology_test.go
  • internal/controller/updater.go
  • internal/controller/updater_test.go
  • internal/metrics/metrics.go
  • internal/metrics/metrics_test.go
  • internal/playground/logs/matcher.go
  • internal/playground/logs/matcher_test.go
  • internal/playground/runner/profile.go
  • internal/playground/runner/profile_registry_test.go
  • internal/playground/scenarios/inventory_40_test.go
  • internal/playground/scenarios/inventory_41_44_test.go
  • internal/playground/scenarios/reader_helpers.go
  • internal/playground/scenarios/s02_planned_switchover.go
  • internal/playground/scenarios/s40_observer_test.go
  • internal/playground/scenarios/s40_reader_data_loss.go
  • internal/playground/scenarios/s41_reader_availability_during_failover.go
  • internal/playground/scenarios/s42_reader_stall_isolation.go
  • internal/playground/scenarios/s43_writable_reader_fence.go
  • internal/playground/scenarios/s44_reader_source_convergence_invariant.go
  • internal/state/matrix.go
  • internal/state/matrix_test.go
  • playground/chaos-scenarios.md
  • test/envtest/reconciler_test.go
📝 Walkthrough

Walkthrough

This change adds first-party read-only replica sites with role-aware validation, per-site Service and MySQL configuration, direct replication-source convergence, safer ordered updates, reader-specific observability, expanded playground topology, and supporting tests and documentation.

Changes

Read-only API and schema

Layer / File(s) Summary
Read-only contracts and CRD schemas
api/v1alpha1/*, config/crd/bases/*, charts/bloodraven/crds/*, examples/*, test/envtest/*
Adds the read-only role, reader lag thresholds, per-site Service and MySQL overrides, convergence status fields, conditional validation, generated deepcopy support, and reader examples.

Controller convergence and reconciliation

Layer / File(s) Summary
Direct source convergence
internal/controller/source_convergence.go, internal/controller/topology.go, internal/controller/runner.go
Confirms the writable primary, checks GTID containment, repoints followers to the active primary, records pending/converged/blocked states, fences writable non-promotable sites, and incorporates convergence into recovery and status.
Per-site configuration and Services
internal/controller/reconciler.go
Creates per-site ConfigMaps, merges protected MySQL settings, creates internal Services, applies site-level Service overrides, computes peer addresses and hashes, and defers unsafe existing Deployment updates.
Ordered updates and backup selection
internal/controller/updater.go, internal/controller/backup_reconciler.go, internal/controller/credentials.go
Processes healthy direct followers sequentially, performs promotable standby handoff when required, uses internal Service hosts, and excludes read-only readers from backup source selection.

Operational validation

Layer / File(s) Summary
Metrics, EndpointSlices, and reader recovery
internal/metrics/*, internal/playground/kube/*, internal/playground/scenarios/s40_*
Adds source-convergence metrics, EndpointSlice flattening helpers, structured log matching, and a reader data-loss auto-clone scenario with endpoint-safety checks.
Multi-site playground and chaos coverage
playground/*, internal/playground/scenarios/*, test/component/*, internal/state/*
Expands playground setup and chaos assertions to three-site topologies, validates reader fencing and convergence, and generalizes recovery expectations to one writable plus N-1 read-only followers.
Documentation and examples
docs/docs/*, playground/chaos-scenarios.md
Documents reader endpoints, source convergence, backup restrictions, cold reclone behavior, service inheritance, monitoring, failover, and N-site operations.

Estimated code review effort: 5 (Critical) | ~120 minutes

Possibly related PRs

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 22.54% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Title check ✅ Passed The title is concise and accurately summarizes the main change: adding read-only reader sites.
Description check ✅ Passed It covers the required summary and testing details, though it uses 'Test plan' instead of the template's Verification heading and omits the observability checklist.
Linked Issues check ✅ Passed The implementation covers the linked issue: read-only role, convergence, exclusions, Services, tests, docs, and playground updates are all present.
Out of Scope Changes check ✅ Passed I don't see clear unrelated scope creep; the extra docs, tests, and playground changes support the reader-site feature.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch megamind/read-only-readers

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds first-class non-promotable read-only reader sites to Bloodraven (API/CRD + controller behavior), including direct-primary replication source convergence, reader-isolated readiness semantics, per-site Service/MySQL config overrides, and updated playground/e2e coverage to validate N-site rollouts.

Changes:

  • Introduce read-only SiteRole with conditional placement validation, per-site Service templates, per-site MySQL config overrides, and reader-specific lag thresholds.
  • Implement and test direct-primary replication source convergence (GTID-gated repoint/restart) plus new source-convergence metrics/status fields.
  • Update playground tooling, scenarios, and documentation for a 3-site default topology (2 candidates + 1 reader) and release-profile coverage.

Reviewed changes

Copilot reviewed 79 out of 80 changed files in this pull request and generated 1 comment.

Show a summary per file
File Description
test/envtest/reconciler_test.go Envtest validation/transition coverage for read-only, serviceTemplate rules, and ordered-update deferral.
test/component/source_convergence_test.go Component test for source convergence after restart (no failover history).
test/component/helpers_test.go Mock MySQL updates to reflect replication source/thread state mutations.
playground/setup.sh Require 3 workers; label nodes for {iad,pdx,reader}; adapt waits and user creation to N sites.
playground/rebuild.sh Derive MySQL deployments from CR sites for sidecar rebuild restarts.
playground/manifests/failovergroup.yaml Add reader site with site-only Service/MySQL overrides and reader lag threshold.
playground/chaos.sh Generalize commands to arbitrary site names; NetworkPolicy partition; node lookup via site label.
internal/state/matrix.go Add read-only role; fence writable non-promotables; exclude readers from core readiness evaluation.
internal/state/matrix_test.go Tests for reader isolation and fencing semantics.
internal/playground/scenarios/s40_observer_test.go Unit tests for scenario-40 observer error handling and transitions.
internal/playground/scenarios/s34_operator_kill_during_wait_replica.go Adjust invariants/messages to “N-1 followers read-only”.
internal/playground/scenarios/s22_replication_status_after_recovery.go Adjust invariants/messages for N-site convergence.
internal/playground/scenarios/s19_reclone_interlock.go Adjust convergence predicate to N-site topology.
internal/playground/scenarios/s18_rapid_cr_spec_changes.go Update scenario assertions to N-site topology.
internal/playground/scenarios/s16_mysql_process_kill.go Update convergence expectations for N-site topology.
internal/playground/scenarios/s13_operator_kill_during_bootstrap.go Update convergence expectations for N-site topology.
internal/playground/scenarios/s12_rolling_update_healthy_state.go Update ordered-update scenario wording/assertions for N sites.
internal/playground/scenarios/s12_old_primary_recovery.go Update reconvergence predicate for N-site topology.
internal/playground/scenarios/s11_total_loss_recovery.go Update total-loss scenario for scaling every site to 0 and N-site reconvergence.
internal/playground/scenarios/s10_full_bootstrap_after_data_wipe.go Update bootstrap reconvergence predicate for N-site topology.
internal/playground/scenarios/s08_gtid_divergence_detection.go Update divergence-cleanup predicate for N-site topology.
internal/playground/scenarios/s06_self_fence_isolated_primary.go Scale all peers (not just one) to isolate primary in N-site topology.
internal/playground/scenarios/s05_operator_kill_during_failover.go Update convergence predicate for N-site topology.
internal/playground/scenarios/s02_planned_switchover.go Add verification that every follower directly replicates from new primary after switchover.
internal/playground/scenarios/inventory_40_test.go Ensures scenario 40 is registered + release-profiled; adds host canonicalization/reader assertions tests.
internal/playground/runner/profile.go Extend release profile with scenario 40.
internal/playground/runner/profile_registry_test.go Update release profile expected count and membership.
internal/playground/runner/executor.go Update cleanup/reconverge wording for N-site topology.
internal/playground/logs/matcher.go Add structured log matcher supporting JSON + slog-text formats.
internal/playground/logs/matcher_test.go Tests structured matcher against JSON and text log lines.
internal/playground/kube/endpointslices.go Add EndpointSlice flattening helpers for service endpoint observation.
internal/playground/kube/endpointslices_test.go Tests EndpointSlice flattening + serving/ready selection behavior.
internal/metrics/metrics.go Add bloodraven_replication_source_state gauge and bounded state list.
internal/metrics/metrics_test.go Verify new replication source state metric is registered.
internal/controller/updater.go Add N-site ordered rollout support via UpdateTarget + sequential follower updates + safe handoff rules.
internal/controller/updater_test.go Tests multi-follower updates, reader exclusion from handoff, unhealthy follower drift retention, and readiness gating.
internal/controller/source_convergence.go Implement GTID-gated direct-primary convergence (repoint/restart), status state, and metrics emission.
internal/controller/source_convergence_test.go Tests multi-follower convergence, divergence gates, failure handling, and suppression conditions.
internal/controller/runner.go Integrate source convergence into status; tighten Ready/Degraded semantics (exclude readers); switch internal service hosts for DSNs and clients; drift inspection behavior tweaks.
internal/controller/runner_test.go Update/extend runner tests for new status semantics and writable non-promotable degradation.
internal/controller/reclone_test.go Allow read-only sites as reclone recipients (with confirmation).
internal/controller/mysql_tls.go Add internal per-site Service name/host helpers.
internal/controller/mysql_tls_test.go Tests internal site service host helper.
internal/controller/credentials.go Dial active site’s internal Service for credential reconciliation; TLS ServerName remains client-facing hostname.
internal/controller/backup_reconciler.go Exclude read-only sites from backup source selection and from explicit overrides.
internal/controller/backup_reconciler_test.go Tests backup source exclusion for read-only and updated host expectations.
internal/controller/backup_job.go Switch backup job MySQL host to internal site Service.
internal/controller/archiver_poller.go Switch sidecar client URLs to internal site Service.
examples/minimal-failovergroup.yaml Example updated to include reader role, templates, and replication thresholds.
docs/docs/playground.mdx Document 3-site playground topology and scenario 40 usage.
docs/docs/planned-failover.mdx Document follower source convergence and reader endpoint gating after planned switchover.
docs/docs/operations.mdx Add reader reclone guidance and blocked convergence troubleshooting.
docs/docs/multi-site.mdx Document roles/terminology, topology rules, and N-site behaviors including readers.
docs/docs/monitoring.mdx Add new replication source state metric reference.
docs/docs/monitoring-prometheus.mdx Add PromQL guidance for reader/source-convergence monitoring.
docs/docs/log-schema.mdx Extend stable log schema for source convergence + GTID fields.
docs/docs/kubectl-plugin.mdx Update reclone docs to cover readers and cold reclone semantics.
docs/docs/known-limitations.mdx Document reader limitations and backup-source exclusions.
docs/docs/failover.mdx Document reader isolation, direct-source convergence, and N-site ordered updates.
docs/docs/examples.mdx Update minimal example description to reflect reader.
docs/docs/crd-reference.mdx Update CRD reference for new fields/validations, internal Services, and status fields.
docs/docs/configuration.mdx Document MySQL config precedence, service inheritance, and internal Services behavior.
docs/docs/backup-overview.mdx Document backup source eligibility exclusions for readers.
docs/docs/app-integration.mdx Document client/internal Services and reader endpoint gating semantics.
config/crd/bases/shipstream.io_mysqlstandbyclusters.yaml Regenerated CRD: add reader role, per-site serviceTemplate/mysqlConf, externalTrafficPolicy, and validations.
config/crd/bases/shipstream.io_mysqlfailovergroups.yaml Regenerated CRD: add reader role, per-site serviceTemplate/mysqlConf, externalTrafficPolicy, and new status fields.
charts/bloodraven/crds/shipstream.io_mysqlstandbyclusters.yaml Chart CRD sync with regenerated bases.
charts/bloodraven/crds/shipstream.io_mysqlfailovergroups.yaml Chart CRD sync with regenerated bases.
api/v1alpha1/zz_generated.deepcopy.go Regenerated deepcopy updates for new fields and pointer copying.
api/v1alpha1/types.go API additions: read-only role, per-site mysqlConf/serviceTemplate, readOnlyMaxLagSeconds, new status fields and validations.
api/v1alpha1/site_helpers.go Helpers for role checks and effective lag thresholds.
api/v1alpha1/site_helpers_test.go Unit tests for new helpers and effective lag behavior.
.github/kind/e2e-calico.yaml Increase kind cluster workers to support 3-site E2E topology.
Files not reviewed (1)
  • api/v1alpha1/zz_generated.deepcopy.go: Generated file

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread internal/playground/logs/matcher.go

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 14

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (2)
test/envtest/reconciler_test.go (1)

485-515: 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win

Verify every site's named MySQL container remains untouched.

The safe-rollout contract covers all existing Deployments, but this checks only dc1 and assumes MySQL is Containers[0]. Capture and compare the named MySQL container image for both dc1 and dc2; otherwise concurrent patching of dc2 can pass unnoticed.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@test/envtest/reconciler_test.go` around lines 485 - 515, Update the
existing-deployment verification in the reconciliation test to cover both dc1
and dc2. Capture each Deployment’s MySQL container image by container name
rather than assuming Containers[0], then assert after reconciliation that both
named-container images remain unchanged.
internal/state/matrix.go (1)

71-102: 🎯 Functional Correctness | 🟠 Major | ⚡ Quick win

Count configured core sites, including StateUnknown.

coreCount currently counts only writable/read-only/unreachable buckets. When all primary candidates are still Unknown, it becomes zero and reports Healthy. Increment the core count by role before state classification so startup/failed observations remain NoPrimary, not healthy.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@internal/state/matrix.go` around lines 71 - 102, Update the observation loop
around coreCount so every configured primary-candidate site increments the
core-site count before state classification, including sites with StateUnknown.
Keep fencing and bucket classification behavior unchanged, and ensure an
all-unknown primary-candidate set reaches the existing non-healthy/NoPrimary
handling instead of the coreCount == 0 Healthy path.
🧹 Nitpick comments (3)
internal/metrics/metrics_test.go (1)

39-54: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Fold this into a table-driven metric-registration test.

This adds another one-off registration test; use a case table so future metrics only require another entry.

As per coding guidelines, “Add table-driven unit tests beside the code they cover, using the existing *_test.go layout under internal/.”

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@internal/metrics/metrics_test.go` around lines 39 - 54, Replace the one-off
TestRegisterIncludesReplicationSourceState with a table-driven registration test
covering replication source state and structured for additional metric cases.
Keep each case responsible for setting up any required labels, gathering the
registry, and asserting the expected metric family, while preserving cleanup
through ReplicationSourceState.DeleteLabelValues.

Source: Coding guidelines

test/envtest/reconciler_test.go (1)

402-413: 🔒 Security & Privacy | 🔵 Trivial | ⚡ Quick win

Assert that administrative Services remain internal-only.

Existence alone would allow an -internal Service to regress to NodePort or LoadBalancer. Assert the expected cluster-internal type and absence of external exposure fields.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@test/envtest/reconciler_test.go` around lines 402 - 413, The Service checks
in the reconciler test should validate that each administrative `-internal`
Service remains cluster-internal, not just that it exists and has an owner
reference. For those services, assert the expected internal `corev1.ServiceType`
and verify external exposure fields such as node ports and load balancer ingress
are absent; leave the existing checks for other services unchanged.
internal/controller/reclone_test.go (1)

93-99: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Fold this role case into a table-driven reclone validation test.

Add the reader variant to the adjacent cold-reclone cases instead of duplicating setup and assertions.

As per coding guidelines, “Add table-driven unit tests beside the code they cover, using the existing *_test.go layout under internal/.”

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@internal/controller/reclone_test.go` around lines 93 - 99, Fold
TestValidateRecloneRequest_ReadOnlyRecipient_OK into the adjacent table-driven
cold-reclone validation test, adding a case that configures the recipient with
SiteRoleReadOnly and uses the correct confirmation token. Remove the standalone
test and reuse the existing table setup and assertion flow.

Source: Coding guidelines

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@internal/controller/reconciler.go`:
- Around line 705-721: Update the settings assembly around
normalizedMySQLSettings and the operator-owned invariant map to remove both
skip-log-bin and disable-log-bin aliases, including underscore forms such as
skip_log_bin, before rendering the final configuration. Ensure group and site
overrides cannot reintroduce these keys, while preserving the enforced log-bin
setting, and add coverage for both underscore and hyphen inputs.
- Around line 1305-1314: Update reconcileSiteService to assign
svc.Spec.LoadBalancerIP from site.LBIP when the effective service type is
LoadBalancer, and clear it for other service types or when the configuration
changes away from LoadBalancer. Add tests covering the reader LoadBalancer path
and clearing the field after a service-type transition.

In `@internal/controller/runner.go`:
- Around line 893-900: Move the source-convergence evaluation in the runner’s
replication-status handling ahead of the nil Replication guard, specifically for
read-only follower states. Ensure probe failures with Replication=nil and
Pending/ProbeFailed still set the Degraded condition when SourceConvergenceState
is not sourceConvergenceConverged, while preserving the existing nil-status skip
behavior for unrelated cases.

In `@internal/controller/source_convergence.go`:
- Around line 145-158: Update the rollback restart calls in the post-stop probe
and source-change failure paths within the surrounding convergence method to use
a fresh bounded context derived from the parent context, rather than the
potentially expired ctx. Create or obtain that rollback context before stopping
replication and pass it to follower.mysql.StartReplica, preserving the existing
divergence check and error returns.

In `@internal/controller/topology.go`:
- Line 1296: Bound each MySQL fencing and safety-probe phase, including the
SetSuperReadOnly call and the regions around the referenced probe blocks, with a
short child context deadline derived from the manager Poll context. Use the
existing normal-polling timeout pattern, pass the bounded context to every probe
in each phase, and ensure the child context is released after the phase
completes so stalled drivers cannot block fencing, failover, or status updates
indefinitely.

In `@internal/controller/updater.go`:
- Around line 229-240: Update the failover flow around failover.Execute so the
authority/status change and onPromoted callback occur immediately after a
successful failover, before CheckReadOnly is called. Preserve the existing error
return for CheckReadOnly failures while ensuring the promoted handoff target is
recorded even when that validation fails.
- Around line 212-220: Update the follower selection loop in the updater handoff
logic to track which followers were successfully processed by the earlier
update/probe phase, and only assign handoff to a non-drifted follower or one
whose update completed successfully; do not reselect a drifted follower solely
because a second requireDirectReplica call succeeds. Add a regression test
covering a candidate whose first probe fails and second probe succeeds,
verifying it is not handed off while stale.

In `@internal/metrics/metrics.go`:
- Around line 97-100: Add stable failover-group identity to
ReplicationSourceState by including namespace/group labels alongside site and
state, then update every metric emitter, test, and documentation reference to
supply and describe those labels consistently. Preserve the existing state-set
semantics while ensuring sites with the same name in different groups produce
distinct series.

In `@internal/playground/logs/matcher.go`:
- Around line 39-71: Update textFieldMatches to match complete
whitespace-delimited key/value tokens, requiring a valid key boundary and
ensuring the value ends before whitespace or the end of the line. Preserve
support for quoted values while rejecting key prefixes and value prefixes such
as oldsource=... and source=auto-clone-old; keep Structured’s JSON matching
unchanged.

In `@internal/state/matrix.go`:
- Around line 73-75: Update the matrix action-building flow around the
writable-site handling and PromotionCandidates population: when any writable
site with a non-promotable role is added to FenceSites, return a
fencing-specific degraded action immediately instead of allowing promotion
candidates to be populated. Preserve promotion only after a subsequent poll
confirms the site is no longer writable or has been successfully fenced.

In `@playground/chaos-scenarios.md`:
- Around line 270-277: Update Scenario 11’s adjacent Verify text to state that
recovery results in one writable site and two read-only follower sites, matching
the three deployments scaled by the injection.
- Line 1073: Update the continuous observer description to avoid claiming
immediate failure: state that s40ContinuousObservation.fail records invariant
violations during scale-down, empty-datadir detection, clone, or catch-up, with
errors reported later by stopAndCheck. Do not imply the active operation is
aborted unless the runner is changed to cancel on failure.

In `@test/envtest/reconciler_test.go`:
- Around line 369-377: Update the owner-reference assertion in the per-site
ConfigMap verification loop to validate that each ConfigMap is controlled by the
expected object using metav1.IsControlledBy(&cm, fg), rather than checking only
OwnerReferences[0].Name. Preserve the existing failure reporting and ConfigMap
lookup behavior.
- Around line 161-173: Strengthen the negative cases in the test loop around
tt.wantOK and k8sClient.Create by asserting that the error is a Kubernetes
schema-validation Invalid error, ideally also verifying the expected field path.
Keep the valid-object assertion unchanged, and ensure unrelated failures such as
namespace setup errors cannot satisfy the invalid-object branch.

---

Outside diff comments:
In `@internal/state/matrix.go`:
- Around line 71-102: Update the observation loop around coreCount so every
configured primary-candidate site increments the core-site count before state
classification, including sites with StateUnknown. Keep fencing and bucket
classification behavior unchanged, and ensure an all-unknown primary-candidate
set reaches the existing non-healthy/NoPrimary handling instead of the coreCount
== 0 Healthy path.

In `@test/envtest/reconciler_test.go`:
- Around line 485-515: Update the existing-deployment verification in the
reconciliation test to cover both dc1 and dc2. Capture each Deployment’s MySQL
container image by container name rather than assuming Containers[0], then
assert after reconciliation that both named-container images remain unchanged.

---

Nitpick comments:
In `@internal/controller/reclone_test.go`:
- Around line 93-99: Fold TestValidateRecloneRequest_ReadOnlyRecipient_OK into
the adjacent table-driven cold-reclone validation test, adding a case that
configures the recipient with SiteRoleReadOnly and uses the correct confirmation
token. Remove the standalone test and reuse the existing table setup and
assertion flow.

In `@internal/metrics/metrics_test.go`:
- Around line 39-54: Replace the one-off
TestRegisterIncludesReplicationSourceState with a table-driven registration test
covering replication source state and structured for additional metric cases.
Keep each case responsible for setting up any required labels, gathering the
registry, and asserting the expected metric family, while preserving cleanup
through ReplicationSourceState.DeleteLabelValues.

In `@test/envtest/reconciler_test.go`:
- Around line 402-413: The Service checks in the reconciler test should validate
that each administrative `-internal` Service remains cluster-internal, not just
that it exists and has an owner reference. For those services, assert the
expected internal `corev1.ServiceType` and verify external exposure fields such
as node ports and load balancer ingress are absent; leave the existing checks
for other services unchanged.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: b69e124e-1447-4e2e-9cc4-4562345036db

📥 Commits

Reviewing files that changed from the base of the PR and between a59f59f and 922745e.

📒 Files selected for processing (80)
  • .github/kind/e2e-calico.yaml
  • api/v1alpha1/site_helpers.go
  • api/v1alpha1/site_helpers_test.go
  • api/v1alpha1/types.go
  • api/v1alpha1/zz_generated.deepcopy.go
  • charts/bloodraven/crds/shipstream.io_mysqlfailovergroups.yaml
  • charts/bloodraven/crds/shipstream.io_mysqlstandbyclusters.yaml
  • config/crd/bases/shipstream.io_mysqlfailovergroups.yaml
  • config/crd/bases/shipstream.io_mysqlstandbyclusters.yaml
  • docs/docs/app-integration.mdx
  • docs/docs/backup-overview.mdx
  • docs/docs/configuration.mdx
  • docs/docs/crd-reference.mdx
  • docs/docs/examples.mdx
  • docs/docs/failover.mdx
  • docs/docs/known-limitations.mdx
  • docs/docs/kubectl-plugin.mdx
  • docs/docs/log-schema.mdx
  • docs/docs/monitoring-prometheus.mdx
  • docs/docs/monitoring.mdx
  • docs/docs/multi-site.mdx
  • docs/docs/operations.mdx
  • docs/docs/planned-failover.mdx
  • docs/docs/playground.mdx
  • examples/minimal-failovergroup.yaml
  • internal/controller/archiver_poller.go
  • internal/controller/backup_job.go
  • internal/controller/backup_reconciler.go
  • internal/controller/backup_reconciler_test.go
  • internal/controller/credentials.go
  • internal/controller/mysql_tls.go
  • internal/controller/mysql_tls_test.go
  • internal/controller/reclone_test.go
  • internal/controller/reconciler.go
  • internal/controller/reconciler_test.go
  • internal/controller/runner.go
  • internal/controller/runner_test.go
  • internal/controller/source_convergence.go
  • internal/controller/source_convergence_test.go
  • internal/controller/topology.go
  • internal/controller/topology_test.go
  • internal/controller/updater.go
  • internal/controller/updater_test.go
  • internal/metrics/metrics.go
  • internal/metrics/metrics_test.go
  • internal/playground/kube/endpointslices.go
  • internal/playground/kube/endpointslices_test.go
  • internal/playground/logs/matcher.go
  • internal/playground/logs/matcher_test.go
  • internal/playground/runner/executor.go
  • internal/playground/runner/profile.go
  • internal/playground/runner/profile_registry_test.go
  • internal/playground/scenarios/common.go
  • internal/playground/scenarios/inventory_40_test.go
  • internal/playground/scenarios/s02_planned_switchover.go
  • internal/playground/scenarios/s05_operator_kill_during_failover.go
  • internal/playground/scenarios/s06_self_fence_isolated_primary.go
  • internal/playground/scenarios/s08_gtid_divergence_detection.go
  • internal/playground/scenarios/s10_full_bootstrap_after_data_wipe.go
  • internal/playground/scenarios/s11_total_loss_recovery.go
  • internal/playground/scenarios/s12_old_primary_recovery.go
  • internal/playground/scenarios/s12_rolling_update_healthy_state.go
  • internal/playground/scenarios/s13_operator_kill_during_bootstrap.go
  • internal/playground/scenarios/s16_mysql_process_kill.go
  • internal/playground/scenarios/s18_rapid_cr_spec_changes.go
  • internal/playground/scenarios/s19_reclone_interlock.go
  • internal/playground/scenarios/s22_replication_status_after_recovery.go
  • internal/playground/scenarios/s34_operator_kill_during_wait_replica.go
  • internal/playground/scenarios/s40_observer_test.go
  • internal/playground/scenarios/s40_reader_data_loss.go
  • internal/state/matrix.go
  • internal/state/matrix_test.go
  • playground/chaos-scenarios.md
  • playground/chaos.sh
  • playground/manifests/failovergroup.yaml
  • playground/rebuild.sh
  • playground/setup.sh
  • test/component/helpers_test.go
  • test/component/source_convergence_test.go
  • test/envtest/reconciler_test.go

Comment thread internal/controller/reconciler.go
Comment thread internal/controller/reconciler.go Outdated
Comment thread internal/controller/runner.go Outdated
Comment thread internal/controller/source_convergence.go
Comment thread internal/controller/topology.go Outdated
Comment thread internal/state/matrix.go
Comment thread playground/chaos-scenarios.md
Comment thread playground/chaos-scenarios.md Outdated
Comment thread test/envtest/reconciler_test.go
Comment thread test/envtest/reconciler_test.go

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick comments (1)
cmd/playground-chaos/reset.go (1)

117-117: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Add regression coverage for the new normalization phase.

Please verify that normalizeFreshBaseline runs after the first healthy baseline, preserves the scale-down → status-clear → restart → wait ordering, and stops on the first error. This path performs several API-server mutations and should be covered before merge.

As per coding guidelines, run make test-envtest when you touch anything that interacts with the API server.

Also applies to: 129-145

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@cmd/playground-chaos/reset.go` at line 117, Add regression tests for the
reset sequence around normalizeFreshBaseline, verifying it runs after the first
healthy baseline, preserves scale-down → status-clear → restart → wait ordering,
and stops immediately on the first error. Cover the API-server mutation
interactions using the existing test patterns and run make test-envtest.

Source: Coding guidelines

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@cmd/playground-chaos/reset.go`:
- Line 117: Add regression tests for the reset sequence around
normalizeFreshBaseline, verifying it runs after the first healthy baseline,
preserves scale-down → status-clear → restart → wait ordering, and stops
immediately on the first error. Cover the API-server mutation interactions using
the existing test patterns and run make test-envtest.

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 150c0194-d8d6-4372-bf2d-8d36af43c7c3

📥 Commits

Reviewing files that changed from the base of the PR and between 922745e and d8d689c.

📒 Files selected for processing (6)
  • cmd/playground-chaos/reset.go
  • internal/controller/updater.go
  • internal/controller/updater_test.go
  • internal/playground/scenarios/s31_pitr_verification_rustfs.go
  • internal/playground/scenarios/s40_observer_test.go
  • internal/playground/scenarios/s40_reader_data_loss.go
🚧 Files skipped from review as they are similar to previous changes (4)
  • internal/controller/updater_test.go
  • internal/playground/scenarios/s40_observer_test.go
  • internal/playground/scenarios/s40_reader_data_loss.go
  • internal/controller/updater.go

Automates the R3/R4/R5/R9 chaos proposals from the issue #115 review
comment as playground scenarios with the standard Precheck/forensics
contract:

- 41-reader-availability-during-failover (R3, release): the reader
  answers SELECTs continuously through an unplanned failover and then
  repoints directly at the new primary; source-host history proves no
  chained or blocked intermediate state.
- 42-reader-stall-no-group-degradation (R4, smoke+release): SOURCE_DELAY
  grows reader lag past readOnlyMaxLagSeconds for a 3x-maxLagSeconds
  soak with zero group-level effect; only endpoint shedding plus lag and
  source-state metrics react.
- 43-writable-reader-fence (R5, release): a writable reader is fenced
  without debounce, a planned failover targeting it is rejected with the
  role error, and its errant GTID trips the convergence containment gate
  into Blocked/GTIDDiverged; cleanup reconciles the errant transactions
  as empty transactions on the primary.
- 44-reader-source-convergence-invariant (R9, release): a manually
  repointed (chained) reader heals back to the direct primary as a
  poll-loop invariant, emitting the documented convergence log events
  with no failover consumed.

Shared reader helpers (topology resolution, serving-status contract,
marker seeding/replication, endpoint waits) move out of scenario 40 into
reader_helpers.go for reuse. R6-R8 remain documented as manual
follow-ups in chaos-scenarios.md.
All four new reader scenarios (41-44) plus the smoke profile were run
against a fresh 3-site k3d playground. Two fixes from that run:

- s02 follower-convergence verify: tolerate a transiently empty
  status.activeSite after PlannedFailoverPhaseSucceeded (the snapshot
  can publish one cycle before writable confirmation lands); only a
  different non-empty site is a terminal failure. This flaked the first
  smoke batch and passed on rerun with the fix.
- Progress messages printed *int64 SecondsBehindSource as a pointer
  address (lag=0x...); add formatLag and use it in scenarios 40-44.

Live results: 44 PASS 3s, 42 PASS 1m33s, 43 PASS 13s, 41 PASS 39s,
then smoke profile (01, 02x2, 42) all PASS.
@colinmollenhour

colinmollenhour commented Jul 18, 2026

Copy link
Copy Markdown
Collaborator Author

AI Review Response · Commit: b7b292d · By: AI agent · Summary

Addressed all 21 review comments (20 unique findings; the Copilot and CodeRabbit log-matcher comments were duplicates) in dd86c11, now included in the current branch head.

Fixes

  • Hardened MySQL and failover safety: protected binary logging, bounded safety probes and rollback contexts, blocked promotion until non-promotable writable sites are fenced, preserved source-mismatch degradation on nil probe results, and corrected ordered-update handoff/authority recording.
  • Corrected reader Service behavior: applied/cleared lbIP across Service type transitions.
  • Scoped replication-source metrics by namespace/group and updated emitters, tests, PromQL guidance, and metric documentation.
  • Made slog text assertions token-exact and added key/value prefix regression coverage.
  • Strengthened envtest coverage for Invalid schema errors, namespace setup, controller ownership, internal-only Services, and both named MySQL containers.
  • Added reset normalization ordering/error-stop coverage, folded the reclone/metric cases into table-driven tests, fixed unknown-core-site handling, and corrected chaos scenario wording/cardinality.

Review disposition

  • Resolved all 15 inline review threads after pushing the fixes.
  • Addressed all six summary-only findings from the CodeRabbit reviews.
  • No rebuttals, skipped threads, or follow-up questions.
  • The generated CodeRabbit walkthrough/docstring warning was informational and did not identify a concrete blocking review defect.

Verification

  • make generate && make manifests
  • make vet
  • make lint
  • make test
  • make test-envtest with CI-equivalent setup-envtest assets

The PR is open with zero unresolved review threads. CI is running for the latest branch head; Generate Check has passed, E2E is skipped, and the remaining jobs are in progress.

@colinmollenhour
colinmollenhour merged commit 3901dcf into main Jul 18, 2026
15 checks passed
@colinmollenhour
colinmollenhour deleted the megamind/read-only-readers branch July 18, 2026 06:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Read-only reader sites: first-party non-promotable replicas (app RO pool + customer ETL/CDC/OLAP)

2 participants